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APOLLO EXPERIENCE REPORT 


PROBLEM REPORTING AND CORRECTIVE ACTION SYSTEM 

ByT. J. Adams 

Lyndon B. Johnson Space Center 
SUMMARY 


The Apollo spacecraft Problem Reporting and Corrective Action System was de- 
signed to ensure that rapid identification and reporting were accomplished and that vig- 
orous analysis and disposition of problems were made before flight. To accomplish 
this goal, various techniques were used and refinements made during the program, re- 
sulting in the present closed- loop approach. Every problem was carefully analyzed, 
and recurrence control was initiated to ensure early maturity of the hardware. A large 
number of open problems existed in the 1965-1966 time frame, and many means were 
used by the NASA Lyndon B. Johnson Space Center (formerly the Manned Spacecraft 
Center) and the contractors to resolve these problems. The fact that these problems 
were resolved and closed in a relatively short period of time is a credit to all concerned. 
Features of the Problem Reporting and Corrective Action System used in the Apollo 
Program are applicable to future manned spacecraft, but care should be exercised to 
adapt the system to the requirements of the new applications. 


INTRODUCTION 


Problem reporting and disposition were of significant importance in achieving 
maturity of the Apollo spacecraft hardware. Recognition of this importance was inade- 
quate in the early phases of the program, resulting in the need for many refinements. 
The present Apollo Problem Reporting and Corrective Action System (PRACAS) is dis- 
cussed and background presented on deficiencies that existed in the early systems and 
the methods used to correct these deficiencies. 

Early in the Apollo Program, the following ground rules were established. 

1. No flight shall be launched with unresolved or unexplained problems. 

2. All problems must be analyzed to establish the cause so that corrective action 
can be taken or the risk of not taking action can be explained. 



3. The closeout criteria must include a documented correction (that is, drawing 
changes, specifications, procedures, processes, and so forth) that is applicable to 
either the hardware, software, or both. 

The present system was designed using these ground rules and has the following 
requirements. 

1. All problems occurring from acceptance tests through flight missions must 
be reported. Problems occurring before acceptance tests must also be reported, but 
formal management review and closeout are unnecessary unless the problem is con- 
sidered critical from a schedule and cost standpoint. 

2. An analysis of the problem is to be made by knowledgeable, design- cognizant 
individuals to ascertain the cause of the problem and to devise an acceptable corrective 
and preventive action. 

3. The analysis and the corrective action taken for each problem are to be tech- 
nically reviewed and verified independently by Reliability and Management personnel 
within the contractor organization. The purpose of this review is to confirm adequate 
technical actions and recurrence control to meet program schedules on all "like" 
hardware. 

4. An independent review is made by Safety, Reliability, and Quality Assurance 
(SR&QA) and Management personnel of the NASA Lyndon B. Johnson Space Center (JSC) 
(formerly the Manned Spacecraft Center (MSC)) to ascertain the technical adequacy of 
the contractors’ problem closeouts and also the adequacy and effectiveness of the 
PRACAS itself. 

5. Continuously updated management visibility of the status of all open problems 
is provided. 

6. The contractors are required to report critical problems promptly to NASA. 

BACKGROUND AND PRESENT SYSTEM DESCRIPTION 


In the early stages of the Apollo Program, contractors and suppliers used their 
existing corporate procedures for problem reporting, analysis, and corrective action. 
The system primarily used led to the use of an electronic data processing (EDP) sys- 
tem as noted in figure 1. The primary feature of the early system was reporting the 
status of failures with the use of the EDP system. The contractors entered the failure 
on computer tapes and updated the tapes at regular intervals until the problem was re- 
solved. All spacecraft and ground support equipment (GSE) failures were required to 
be entered. Individual failures were required to be closed within 30 to 45 days after 
occurrence. Copies of the tapes were provided to NASA, and weekly printouts were 
made and distributed to cognizant MSC personnel. Appropriate NASA personnel re- 
viewed the tape printouts and provided concurrence or nonconcurrence with the con- 
tractor's proposed closeout. Copies of the tape printout containing the NASA comments 
were provided the contractor for his information or action. Printout summaries of the 
tapes were provided periodically to NASA management. It was never possible to obtain 
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Figure 1. - Early Failure Reporting and Corrective Action System. 


current information utilizing the tapes. The NASA technical personnel were able to 
maintain current information by working with their contractor counterparts. 

Critical failures were reported to NASA within 24 hours of occurrence. Critical 
failures were defined as those occurring during qualification/certification test and those 
impacting schedules, cost, or launch (program impact failures). 

This system existed from the beginning of the program without significant change 
until 1966 when two changes were made. First, the requirement for reporting all GSE 
failures to NASA was changed to reporting only those failures of GSE used in countdown 
and of other GSE that had a safety or spacecraft impact. This change was made because 
of the large number of GSE failures and the need to emphasize those failures requiring 
recurrence control. This requirement, however, did not relieve the contractor of the 
responsibility for continuing to analyze and close all GSE failures. Second, the con- 
tractors prepared an Apollo Problem Summary (APS) of each problem for use at the 
Flight Readiness Review (FRR). The problem summaries served two purposes: 

(1) they summarized major program impact problems for the FRR board, and (2) they 
explained problems that would not be corrected before flight. 

It became apparent in late 1966 and early 1967 that several deficiencies existed 
in the system that required correction. Centralization of the contractors' failure re- 
porting and corrective action system was needed. Failures were not accumulated at or 
managed from a central location. This resulted in the loss of reported failures; inade- 
quate awareness of the status of problems by contractor and NASA management; 
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inadequate support of failure closeouts to meet program milestones; and poor use of 
available resources. In addition, with various Apollo spacecraft being fabricated during 
this time frame, it was mandatory to have better control of the effectiveness of changes 
resulting from failure closeout. As a result, centralized problem assessment areas 
were established. All failures were reported to these areas. Representatives from 
all disciplines concerned with failure closeout were assigned to these areas. Time 
lines were established for each failure to show vehicle and GSE closeout effectivity to 
support vehicle milestones. These time lines were displayed to provide management 
awareness. 

Many problems that existed were significant but did not fall within the definition 
of a failure and therefore were not reported or tracked within the overall system. Such 
items as dented tanks, contamination, and so forth, were included in this category and 
were considered unsatisfactory conditions rather than failures. These types of prob- 
lems were handled by the contractors' internal system but they were not reported to 
NASA even though they were significant. Accordingly, the name and content of the 
Failure Reporting and Corrective Action System was changed to Problem Reporting and 
Corrective Action System to encompass both failures and unsatisfactory conditions. 

(The PRACAS nomenclature is defined in the appendix. ) 

The Resident Apollo Spacecraft Program Office (RASPO) at each prime contractor 
plant was the prime interface with the contractor for problem closeouts. These resi- 
dent personnel were located in the contractor problem assessment areas and were re- 
quired to concur in all problem closeouts. This permitted a more timely review of 
contractor activities and a more timely NASA concurrence of closeouts. Coordination 
was accomplished with MSC on all problems. 

To improve reporting of problems from the prime contractor suppliers, quality 
assurance (QA) delegations to Government agencies in residence at the supplier plants 
were amended to require reporting of significant problems to MSC. This provided a 
check on the adequacy of supplier reporting to the prime contractor, although, in some 
cases, the response from the Government agency was inadequate. 

In late 1967, after repeated attempts to streamline the EDP system to make it 
more timely, the contractors finally abandoned it as a real-time system and began re- 
porting all problems to MSC by datafax through the local RASPO. The open problem 
list (OPL) was instituted by MSC, at first manually and then by automated printout, to 
track open problems. The MSC RASPO was required to approve all contractor problem 
closeouts. The EDP system was still used, but only as a data bank for the history of 
problems. Personnel at the launch site were required to report problems in real time 
to the prime contractor. In addition, launch site personnel were given copies of the 
problem closeout packages on problems reported from the launch site and copies of the 
OPL. At this time, the Apollo Configuration Change Board began to review chronic 
open problems to provide management incentive for closeout. A procedure entitled 
"Explained Problems" was added to the system and was presented to the FRR board 
before each mission. These problems were understood but were not closed. However, 
the failure mode and its effect were understood and sufficient information was developed 
to justify the risk, if any. 
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With these improvements, PRACAS stabilized. The contractor portion of the 
system is shown in figure 2(a) and the internal MSC interfaces in figure 2(b) . The sys- 
tem is summarized in the following sections. 



(a) Contractor flow. 


Approval/disapproval, 



(b) Internal MSC interface. 


Figure 2. - Present Problem Reporting 
and Corrective Action System. 


Reporting of Significant Problems 

The contractors were required to re- 
port, within 24 hours of occurrence or de- 
tection, all problems that occurred during 
or after acceptance testing that could ad- 
versely affect safety, contribute to schedule 
delay, or result in design change; also all 
problems that occurred during certification 
testing or while setting up equipment for 
acceptance tests. If recurrence of the prob- 
lem on like hardware had safety implications, 
the hardware supplier was required to pro- 
vide recommendations for usage restrictions 
until problem analysis and resolution were 
complete. Also, the hardware supplier was 
required to forward to MSC problem reports 
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received from subtier hardware suppliers within 24 hours of receipt. The report con- 
tained, at a minimum, the data shown in figure 3. Within 1 calendar week of occur- 
rence or detection, the additional data shown in the second column of figure 3 were 
reported to MSC. 


Reporting of Routine Problems 

The contractors were required to report, within t calendar week of occurrence or 
detection, routine problems that occurred during or after acceptance testing that would 
not adversely affect the program. The minimum data reported are shown in figure 3; a 
sample of one format used is shown in figure 4. Reporting continued for the operational 
life of the equipment. 


Storage and Retrieval File 

A permanent storage and retrieval file of problems was maintained by MSC based 
on EDP tapes submitted by the contractors. This file was used for various types of en- 
gineering investigations based on problem history. 


Immediate Notification 

Incoming problem reports were reviewed by MSC to determine which ones should 
be brought to the immediate attention of program management. A "Problem Notifica- 
tion" form, shown in figure 5, was completed for each problem thus categorized. Prob- 
lem notifications were categorized into one of three groupings: "STD" (standard) for 

those to be distributed to technical personnel such as subsystem managers and technical 
monitors but not to program management, "Management" for those that required dis- 
tribution to MSC program management and subsystem managers and technical monitors, 
and "Flash" for those recommended for forwarding by MSC program management to 
NASA Headquarters. The same distribution was made for "Flash" notifications as for 
"Management. " Each notification was marked "N" for "noncritical" or "p" for 
"program. ” 


Routine Notification and Problem Status 

A central point at MSC managed all problem data. Distribution of data, including 
copies of problem reports and resolution information, was made by this central point. 
The hardware supplier sent to MSC all reportable problems, including those for which 
a 24- hour report had been submitted. Real-time displays of open problem data affecting 
the next scheduled manned spacecraft launch were maintained by the central point for 
use by the program manager. The hardware supplier periodically submitted to MSC a 
listing of all open problems. Periodic listings of open problems were published and 
distributed by the central point. These listings provided problem status information 
and indicated the applicability of the problems to particular spacecraft. Commencing 
60 days before each manned spacecraft launch, a chart was prepared by the central 
point showing the number of problems that were considered applicable to the vehicles 
or supporting equipment for that mission. The chart was prepared weekly and updated 
daily after the Headquarters FRR and distributed to appropriate MSC program manage- 
ment. An example of the chart is shown in figure 6. 
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DATA ELEMENT MATRIX 


The following matrix indicates elements of data that the hardware supplier shall, as a minimum, 
report toMSCfor each nonconformance. 


The hardware supplier shall record, and retain 
for a time prescribed by contract, all relevant 
data for each nonconformance. 


^ / jf/ i 


Uniquely Identifiable Report Number 

NO 

YES 

YES 

YES 

■L 

YES 

Date of nonconformance occurrence, or date 
nonconformance was detected if occurrence 
date is indeterminable 

YES 

YES 

YES 

YES 

YES 

Indication of whether nonconformance is 
classified as failure or unsatisfactory 
condition 

YES 

YES 

YES 

YES 

YES 

Part number on which nonconformance occurred 

YES 

YES 

YES 

YES 

YES 

Part name on which nonconformance occurred 

YES 

YES 

YES 

YES 

YES 

Serial number of part on which nonconformance 
occurred 

YES 

YES 

YES 

YES 

YES 

Manufacturer of part on which nonconformance 
occurred 

NO 

YES 

YES 

YES 

YES 

Symptom of nonconformance 

YES 

YES 

YES 

YES 

YES 

Test being performed at time of occurrence 

YES 

YES 

YES 

YES 

YES 

Brief, narrative description of nonconformance 

YES 

YES 

YES 

YES 

YES 

End item on which nonconformance occurred, 
if applicable 

YES 

YES 

YES 

YES 

YES 

Prevalent conditions at time of occurrence 

YES 

YES 

YES 

YES 

YES 

All end items which may be affected by 
nonconformance 

NO 

YES 

YES 

YES 

YES 

Problem report numbers, and dates, that 
relate to the same problem 

NO 

NO 

YES 

YES 

AA 

Criticality with relationship to mission 
effects (see Attachment C) 

YES 

YES 

YES 

YES 

YES 

Indication of whether nonconformance is design 
oriented or manufacturing oriented 

IK 

IK 

YES 

YES 

AA 

Analysis results, including laboratory test 
results 

NO 

IK 

YES 

t 

AA 

Cause of nonconformance 

IK 

IK 

YES 

X 

AA 

Corrective action 

NO 

IK 

YES 

N/A 

AA 

Planned date of dispositioning 

NO 

YES 

NO 

NO 

YES 

Explanation rationale 

N/A 

N/A 

NO 

YES 

NO 

Assurance that explanations using redundancy 
and/or alternate modes of operation as one of the 
elements do not negate each other 

NO 

NO 

NO 

YES 

NO 

When last test of article, prior to mission, is to 
be performed. Statement as to whether or not non- 
conformance is detectable during mission 

NO 

NO 

NO 

YES 

NO 

Effect on mission if nonconformance recurred and 
recommended operational workaround procedures 

NO 

NO 

NO 

YES 

NO 

Previous history of nonconforming article 

NO 

NO 

YES 

YES 

NO 


X Hardware supplier shall indicate any f 

AA As Available, prior to resolution 

N/A Not Applicable 

IK If Known 


Figure 3. - Example of data element matrix form. 
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NASA - MANNED SPACECRAFT CENTER 
FAILURE INVESTIGATION ACTION REPORT 


NO. 


1. PROJECT 

2. WHERE DETECTED 

3. ORG. REPORT NO. 

4. PROB. CLASSIF. 

5. DATE REPORTED 


FACILITY 

ORGANIZATION 

LOCATION 


□ FAILURE 
n UNSAT. COND. 



6. CONTRACTOR 

7. END ITEM NAME 

8. ITEM UNDER TEST 

9. NEXT ASSY. NAME 

10. REPORTED ITEM 

11. TPS NUMBER 

7a. El MODEL NO. 

8a. CONTR. PART NO. 

9a. CONTR. PART NO. 

10a. CONTR PART NO. 

12. ROUTING VIA 

7b. El SERIAL NO. 

8b. SUPPLIER PART NO. 

9b. SUPPLIER PART NO. 

10b. SUPPLIER PART NO. 

1 13. SPEC PROCESS NO. 

| DATE: PARA: 

8c. SERIAL NO. 

9c. SERIAL NO. 

10c. SERIAL NO. 

14. COND. 15. CAUSE 16. SYMPT 17. FAIL TYP 18. DETECTED 19. 

DURING 

20. SYSTEM NAME 

lOd. Time Cycles (ACUM) 


21. DESCRIPTION OF FAILURE CONDITION 


22. CRITICALITY 


23. INITIATOR CONTACT ORG- 


25. HARDWARE ANALYSIS REQUESTED INSTRUCTIONS 


DATE 


24. RIE 


ORG. 


DATE 


26. ASSIGNED TO ORG. 


28. CAUSE OF FAILURE ANALYSIS RESULTS 


DATE 


27. REQUESTER 


ORG. 


DATE 


29. SYSTEM ENGINEER ORG. 


31. CORRECTIVE ACTION REQUESTED 


DATE 


30. RIE 


ORG. 


DATE 


32. ACTION ASSIGNED TO ORG. 


34. CORRECTIVE ACTION TAKEN 


DATE 


33. REQUESTER 


ORG. 


DATE 


35. ACTION BY ORG. 


MSC FORM 2174 (JUL 66) 


DATE 


36. RIF 


ORG 


DATE 


37. CLOSE-OUT 


DATE 


PAGE OF 


Figure 4. - Example of failure investigation action report. 
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□ FAILURE □ 

UNSAT COND. □ UNDETERMINED 


PROBLEM NOTIFICATION 

SSM 

FLASH STD.. MSC LOG NO. 

SSE 

MANAGEMENT REPORT NO. 

O CSM □ LM □ G8>C 

PAE ASSESSMENT N P PAC CODE (NR) 

ORIGINATOR 

LOCATION . . PHONE 

NONCONFORMANCE SITE 

S/C OCCURRED ON OR FIRST AFFECTED 

NONCONFORMANCE OCCURRED DURING 

DATE OF NONCONFORMANCE 

SUBSYSTEM 

PATE/TIME RECEIVED 

CRITICALITY 

CONTRACTOR 

SUBCONTRACTOR 

PART NUMBER/SN 

PART NAME 

NEXT HIGHER ASSEMBLY 

TP NO. 

NONCONFORMANCE DESCRIPTION 



SUSPECTED CAUSE 


DISTRIBUTION STANDARD, PLUS 


RECEIVED BY 


PROBLEM ASSESSMENT ENGINEERING 


Figure 5. - Example of problem notification report. 
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MISSION-OPEN PROBLEM TRENDS 



10 



Problem Closeout 


The MSC was required to concur in all contractor problem closeouts. The prob- 
lem control sheet shown in figure 7 was used to record MSC acceptance of closeouts. 
Signatures were required by the cognizant MSC design engineer and the SR&QA engineer. 
The problem was closed or explained. Contractors were provided copies of the prob- 
lem control sheets. The OPL was updated to reflect the actions taken, such as complete 
removal of a problem if it was resolved for the entire program or indication of the 
spacecraft to which the resolution was applicable. 


Potential Hardware Impact Problems 

Those problems that were considered to have a potential hardware impact were 
marked on the OPL with the letters "PHI. " If known, the date the nonconformance was 
deemed to have a potential hardware impact was also noted. When applicable, the end- 
items affected were indicated. 


Management Reviews 

A series of selected contractor and customer management reviews was conducted 
at various levels of management throughout the program to discipline the system and 
accelerate problem closures. 

The Apollo hardware problem experience from program initiation to mid-1972 is 
shown in figure 8. More than 50 000 problems were experienced during the course of 
the Apollo Program. The slope of the cumulative problem curve is very sensitive to 
program activity. The peak in problems occurred in early 1967 (peak of certification 
test activity), and a gradual slowdown in activity occurred in early 1969 (completion of 
most of the certification test activity and subsystem deliveries). 

An evaluation was made of various problem causes. The results are plotted in 
figure 8(b) for the Apollo spacecraft and in figure 8(c) for the Apollo spacecraft ground 
support equipment. 

From figure 8(b), it can be calculated that more than 18 percent of the Apollo 
spacecraft problems were from design causes; more than 35 percent were due to 
manufacturing/procedure causes; and approximately 20 percent were due to human er- 
ror. Similar calculations for ground support equipment can be obtained from figure 8(c). 

Another significant item evident in figure 8(b) is that, although the majority of the 
spacecraft design and human error problems occurred in 1966, the manufacturing/ 
procedure problems continued. This is explained, in part, because the major part of 
the certification test program was completed and a trained checkout team was fully op- 
erational by 1967. However, even though manufacturing was significantly reduced during 
this period, a large amount of rework (because of design changes) added substantially to 
the manufacturing and testing level of effort. This may explain the higher incidence of 
manufacturing/procedure problems. The problems experienced from 1963 to 1972 for 
various Apollo hardware (command and service module (CSM), lunar module (LM), 
guidance and navigation (G&N), and Government furnished equipment (GFE)) are shown 
in figures 9(a) to 9(g). The figures again emphasize peak problem activity in 1966. 
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Figure 7. - Example of problem control sheet. 
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60 x 10 3 



(a) Total. 



(a) Total. 




(b) Spacecraft. (b) Lunar module. 




(c) Ground support equipment. 


(c) The LM ground support equipment. 


Figure 8. - Apollo hardware cumulative 
problem experience. 


Figure 9. - Apollo hardware problem 
experience. 
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(d) Command and service module. 


(f) Guidance and navigation. 




(e) The CSM ground support equipment. (g) Government furnished equipment. 

Figure 9.- Concluded. 


Subsystem breakdowns denoting where the problems were detected are shown in 
table I. These subsystems are listed in order by percentages of total spacecraft hard- 
ware failures found in each subsystem. The majority of the failures were in the elec- 
trical subsystems; the percentages found in screening tests were especially high. 
Because of difficulties in obtaining components that would meet mission requirements, 
special attention was given to an electrical, electronic, and electromechanical (EEE) 
parts evaluation program. In the primary mechanical subsystem, the overall failure 
rates and the percentages of screening test failures were relatively low, while the per- 
centages of problems found in certification testing were generally greater than the mean. 
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TABLE I. - SUBSYSTEM FAILURE DETECTION 


[Data as of September 1972] 


Subsystem 

— 

Total failures, 
percent 

Subsystem failures breakdown, 
percent 

A TP/PI T a 

Certification 

test 

Vehicle 

Preflight 

Inflight 

(b) 

CSM 

Instrumentation 

15.9 

77.3 

3.9 

18.4 

0.4 

Environmental control 

13.9 

72. 5 

12.2 

13.9 

1.4 

Displays and controls 

12.4 

71.9 

10.2 

17.2 

.7 

Propulsion 

10.8 

72.4 

19.4 

8.1 

.1 

Reaction control 

10.4 

60.2 

15. 3 

24.0 

. 5 

Electrical power 

7.6 

55. 7 

24. 5 

18.8 

1.0 

Communications 

7.4 

68.1 

18.9 

11.6 

1.4 

Stabilization control 

4.8 

77.8 

15.1 

5.9 

1.2 

Fuel cell and cryogenics 

4.7 

66. 0 

22.7 

9.7 

1.6 

Guidance and control 

3.9 

61. 5 

16.4 

19.5 

2.6 

Structures and mechanics 

2.8 

36.2 

36.0 

26. 5 

1.3 

Earth landing and uprighting 

2.0 

46.9 

31.6 

19.6 

1.9 

Crew equipment 

1.8 

34. 7 

51.2 

12.1 

2. 0 

Ordnance 

1.6 

46.0 

44.7 

8.7 

.6 

Mean percent 


60.6 

22.9 

15.3 

1.2 

LM 

Electrical assemblies 

18.6 

74. 0 

16.0 

9. 5 

0.5 

Propulsion 

15.1 

69.9 

11.4 

18.6 

.1 

Instrumentation 

12. 5 

59.0 

13.8 

26. 7 

. 5 

Environmental control 

9.1 

62.6 

14.3 

22.7 

.4 

Radar 

7.6 

76.4 

15.4 

8.2 

.0 

Reaction control 

7.6 

57. 5 

13.9 

27.5 

1.1 

Communications 

6.9 

77.2 

16.8 

5.9 

.1 

Abort guidance 

6.2 

68.2 

16.1 

15. 3 

. 4 

Stabilization control 

5.7 

66. 5 

25. 3 

7.7 

. 5 

Electrical power 

4.3 

37. 3 

26.8 

35. 3 

.6 

Structures and mechanics 

4.1 

32.9 

24.4 

42. 5 

.2 

Crew equipment 

2.2 

42. 5 

33.6 

23. 3 

.6 

Mean percent 


60. 3 

19.0 

20. 3 

.4 


Acceptance test procedure/preinstallation test. 
^Includes postflight failures for CSM only. 
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PERFORMANCE OF THE PROBLEM REPORTING 
AND CORRECTIVE ACTION SYSTEM 


As can be seen in figure 9(a), the rate of problems occurring in the program rose 
very steeply in 1965 and 1966, peaking at 1800 per month in late 1966. To handle this 
quantity, several means were used by NASA and the contractors to close the problems 
expeditiously. Handwritten duplicate logs of open problems were maintained at MSC and 
at the prime contractor plants to track the open problems. Daily telephone conferences 
were held between cognizant NASA and contractor engineering personnel to discuss open 
problems and develop means for closeouts. On critical problems, special teams of 
NASA and contractor personnel were established to work the problem in "real time. " 
This involved, in some cases, hand- carrying the failed hardware to the vendor for fail- 
ure analysis, witnessing the failure analysis, and expediting the paperwork for problem 
closeout. As a result of these efforts and the corrective actions leading to hardware 
maturity, the number of problems dropped dramatically in late 1966. 


CONCLUDING REMARKS 


The Apollo Program was a very complex program undertaken by NASA. The size 
of the program dictated that a new approach was necessary to understand and correct 
problems. This was not initially recognized, and a series of changes took place to cor- 
rect the deficiencies in the Problem Reporting and Corrective Action System to develop 
it into its present form. The rate of occurring problems was very high from mid-1965 
to late 1966. Means used to evaluate and close this large number of problems included 
handwritten tracking logs, daily telephone conferences between cognizant NASA and con- 
tractor personnel, and formation of special task teams to work on critical problems. 

The fact that such a large number of problems were closed is an achievement worthy 
of note. Features of the Problem Reporting and Corrective Action System used in the 
Apollo Program are applicable to future manned spacecraft, but care should be exer- 
cised to adapt the system to the different requirements. 


RECOMMENDATIONS 


One recommendation for the design of the Problem Reporting and Corrective 
Action System for future programs is to change the philosophy of requiring that every 
open problem be resolved before flight. For priority of work purposes, it may be pos- 
sible to categorize the problems by the criticality of the equipment involved and use the 
Apollo explain technique. On less critical equipment, problem analysis related to actual 
teardown of hardware may be dictated by trends of occurrence rather than by analysis 
of each problem as it occurs. 

Another possibility is the establishment of a problem analysis facility at the launch 
site. In the Apollo Program, most of the failed hardware was returned to the vendor for 
failure analysis, with attendant delays in shipping and in ensuring that an adequate analy- 
sis was performed by the vendor. This may not be feasible on future programs. 
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There are possible applications of the Apollo spacecraft Problem Reporting and 
Corrective Action System to future manned programs other than those outlined. Many 
of the Apollo guidelines can be applied to ensure that the hardware launched as a part 
of the United States space program is adequate for mission performance. 


Lyndon B. Johnson Space Center 

National Aeronautics and Space Administration 
Houston, Texas, November 5, 1973 
951-18-00-00-72 
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APPENDIX 


DEFINITIONS 


Words or terms used in this report that have a special connotation or meaning are 
listed below. 

Nonconformance: Nonconformance is a condition of any article, material, or 
service in which one or more characteristics do not conform to requirements. This 
includes failures, unsatisfactory conditions, discrepancies, deficiencies, defects, and 
malfunctions. 

Problem: A problem is any nonconformance that fits, or is suspected of fitting, 
into one of the following categories: (1) failure or unsatisfactory condition occurring 

during or after production acceptance testing or (2) failure or unsatisfactory condition 
occurring before acceptance testing that will or has the potential to affect safety ad- 
versely, contribute to schedule impact, cause a launch delay, or result in design change. 

Failure: Failure is the inability of a system, subsystem, component, or part to 
perform its required function within specified limits under specified conditions for a 
specified duration. 

Unsatisfactory condition: An unsatisfactory condition is any defect for which en- 
gineering disposition is required and which requires recurrence control beyond the 
specific article under consideration. Included in this definition are conditions that can- 
not be corrected to the specified configuration using the standard planned operations or 
events that could lead to a failed condition but do not affect the function of the article, 
such as contamination, corrosion, workmanship requiring engineering disposition, and 
so forth. 

Open problem: An open problem is a problem for which responsible MSC manage- 
ment personnel have not approved the problem resolution submitted by the hardware 
supplier. A problem is deemed to be open until the hardware supplier is formally 
notified by MSC that resolutions are acceptable for all deliverable end-items to which 
the problem is applicable. 

Resolved problem: A resolved problem is a problem that has been closed or 
explained. 

Routine problem: A routine problem is a problem that has no potential to affect 
the program adversely. 

Closed problem: A problem is closed when the hardware supplier is formally 
notified of MSC concurrence with the problem analysis (including determination of the 
cause) and with the implementation of corrective action to preclude recurrence of the 
problem on hardware after acceptance tests. A lack of corrective action may be ac- 
ceptable to MSC if analytical and test evidence from the hardware supplier shows that 
the problem is always detectable during the performance of an established test before 
end use, and that the problem will not occur after this test. 
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Explained problem: A problem is explained when the hardware supplier is for- 
mally notified of MSC concurrence with the problem analysis and rationale for not es- 
tablishing corrective action. The rationale must establish that a planned mission may 
proceed with no detrimental effects should the problem recur and that a MSC contrac- 
tually responsible authority (that is, the Configuration Control Board) has decided that 
no corrective action, as defined for a closed problem, need be established. 

Problem analysis: A problem analysis is documented results of the investigation 
performed to determine the cause of the problem. 

Cause (problem cause): The cause or problem cause is the event or series of 
events directly responsible for the problem. 

Corrective action: Corrective action is action established to prevent recurrence 
of the problem. 


NASA -Langley, 1974 


S-384 


19 


